Segmenting A Sentence Into Morphemes Using Statistic Information Between Words

This paper is on dividing non-separated language sentences (whose words are not separated from each other with a space or other separaters) into morphemes using statistical information, not grammatical information which is often used in NLP. In this paper we describe our method and experimental result on Japanese and Chinese se~,tences. As will be seen in the body of this paper, the result shows that this systent is etlicient for most of tile sentences. 1 I N T R O D U C T I O N A N D M O T I V A T I O N An English sentence has several words and those words are separated with a space, i t is e~usy to divide an English sentence into words. I[owever a a apalmse sentence needs parsing if you want to pick up the words in the sentence. This paper is on dividing non-separated language sentences into words(morphemes) without using any grammatical information. Instead, this system uses the stat is t ic information between morphenws to select best ways of segmenting sentences in nonseparated languages. Thin ldng about segmenting a sentence into pieces, it is not very hard to divide a sentence using a certain dictionary for that . The problem is how to decide which ' segmenta t ion ' the t)est answer is. For examl)le , there must be several ways of segmenting a Japanese sentence wri t ten in ll iragana(Jal)a,lese alphabet) . Maybe a lot more than 'several'. So, to make the segmenting system useful, we have to cot> sider how to pick up the right segmented sentences from all the possible seems-like-scgrne, nted sentences, This system is to use statist ical inforn,ation between morphemes to see how 'sentence-like'(how 'likely' to happen a.s a sentence) the se.gmented string is. To get the statist ical association between words, mutual information(MI) comes to be one of the most interesting method. In this paper MI is used to calculate the relationship betwee.n words found ill the given sentence. A corpus of sentences is used to gain the MI. 'Fo implement this method, we iml)lemented a system MSS(Morphological Segmentat ion using Statistical information). W h a t MSS does is to find the best way of segmenting a non-separated language, sentence into morphemes without depending on granamatieal information. We can apply this system to many languages. ~2 ) / [ O R P H O L O G I C A L A N A L Y S I S 2 . 1 W h a t ; a M o r p h o l o g i c a l A n a l y s i s I s A morpheme is the smallest refit of a str ing of characters which has a certain linguistic l/leaning itself. It includes both content words and flmction words, in this l)aper the definition of a morl)heme is a string of characters which is looked u I) in tile dictionary. Morphoh)gical analysis is to: l) recognize the smallest units making up tile given sentellce if the sentence is of a l |on-separated hmguage, divide the sentence into morphenms (automatic segmentat ion) , and 2) check the morlflmmes whether they are the right units to make up the sentence. 2 . 2 S e g m e n t i n g M e t h o d s We have some ways to segment a non-separated sentence into meaningflll morphemes. These three methods exl)lained below are the most popular ones to segment ,I apanese sentences. • T h e l o n g e s t s c ' g m e n t m e t h o d : l~,ead the given sentence fi'om left to right and cut it with longest l)ossible segment. For exampie, if we get ' isheohl ' first we look for segments wilich uses t h e / i r s t few lette,'s in it , ' i ' and 'is'. it is ol)vious tha t 'i';' is loIlger thall ' i ' , SO tile system takes 'is' as the segment. Then it tries the s;tllle method to find the segnlents in 'heold' and tinds 'he' and 'old'. The, least-bunsetsu s e g m e n t i n g m( ' , thod: Get all the possible segmentat ions of the input sentence and choose the segmentat ion(s) which has least buusetsu in it.. ' l 'his method is to seg:ment Japanese sentence.s, which have content words anti function words together in one bunsetsu most of the time. This method helps not to cut a se, ntenee into too small meaningless pieces. Let tm'ty l )e , s e g m e n t i n g m e t h o d : In Japanese language we have three kinds of letters called Iliragana, Katakana and Kanji. This